Recitation Review of Parameter Estimation 10 - 701 / 15 - 781 Machine Learning , Fall 2009 Shay

This is a set of notes, summarizing what we talked about in the second recitation. They are not meant to be rigorous, and are written informally, following what was discussed in class. The Likelihood Function Consider the probability distribution for the following unfair coin-toss random variable: P (X = 0) = 1/3 P (X = 1) = 2/3 This is an example of a probability distribution, and we are used to think about it as a function of possible outcomes of the coin, or other measurement that we have in the realworld. But what if we ask a different question: instead of asking “what is the probability of getting a tail or head,” we ask: “if we don’t know what is the bias of the coin, can we get a probability distribution which will give us some clue about its bias from what we observe?” What we do, more formally, is instead of having a single probability distribution, we define a family of distributions where the above P in Eq.1 is just an instance of. So, one natural family for unbiased coin tosses we could define is: {Pθ | θ ∈ [0, 1]} where Pθ(X = 0) = θ and Pθ(X = 1) = 1− θ. Now, our job is to assume we observe a few coin tosses and decide what is the best Pθ(·) from the above family. Notice, that with “parameteric methods”, we will always be seeking for Pθ(·) where θ ∈ Θ, and seeking that Pθ(·) is equivalent to choosing a θ. There are other families of models which don’t “parameterize” our distributions so easily, we will discuss some of these “non-parametric” ideas later in class. So, keep in mind our goal. It is to find some θ which conforms to what we observe. In order to do that, we change our view of P , and instead of looking at it as a function of X, we look at it as a function of θ, where X is fixed to our observations. More formally, what we do is to define the “likelihood function:” L(θ;x1, ..., xn) = n ∏